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1  Documentation  Roadmap 

This  Final  Report  is  organized  into  the  following  sections: 

•  Section  1  (“Documentation  Roadmap”)  provides  information  about  this  document  and  its  intended 
audience.  It  provides  the  document  overview  and  describes  the  content  of  each  section. 

•  Section  2  (“Program  Overview/Review”)  provides  a  summary  of  schedules  and  milestones 
achieved  during  the  year  by  phase. 

•  Section  3  (“Option  Year  Technical  Overview”)  a  technical  overview  of  the  option  year  effort,  from 
an  architectural  and  implementation  perspective.  Interaction  with  other  ONR  performers  and  testing 
accomplished  are  also  summarized. 

•  Section  4  (“Results,  Conclusions  and  Next  Steps”)  summarizes  overall  status,  lessons  learned  and 
presents  goals  and  objectives  for  the  second  option  year. 

•  Section  5  (“Acronym  List”)  provides  an  acronym  list. 

1 .1  Document  Management  and  Configuration  Control  Information 

•  Revision  Number:  - 

•  Revision  Release  Date:  08/20/10 

•  Purpose  of  Revision:  Initial  Release 

•  Scope  of  Revision:  N/A 

1.2  Purpose  and  Scope 

This  Final  Report  provides  a  summary  of  work  done  under  the  ONR  Transparent  Urban  Struc¬ 
tures  contract  in  the  first  Option  Year  of  the  program.  The  team  is  comprised  of  Textron  Defense 
Systems,  Natural  Selection  Inc,  the  U.S.  Army  Engineering  Research  and  Development  Center 
(ERDC),  and  anthropologist  Liam  Murphy. 

The  first  six  weeks  of  the  contract,  the  team  was  integrating  and  executing  the  ISR-C2  demon¬ 
stration  at  the  MEC  in  Kaneohe,  Hawaii.  Section  3.1  will  provide  a  brief  overview  and  lessons 
learned  from  this  exercise.  These  lessons  learned  fueled  the  development  effort  that  followed, 
leading  through  the  Empire  Challenge  2010  (EC  10),  Green  Devil  II  (GDII)  exercise  at  Ft  Hua- 
chuca  in  August  2010. 

The  ensuing  development  effort  was  a  continuous  effort,  not  designated  into  phases.  However, 
upon  receiving  details  and  direction  regarding  Empire  Challenge  2010  (EC  10),  most  efforts  were 
geared  toward  that  goal  as  we  progressed  through  the  program.  This  document  will  provide  a 
summary  of  activities,  architecture,  functionality  and  initial  results  from  the  EC10/GDII  exercise. 
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2  Program  Overview/Review 


This  section  reviews  programmatics. 

2.1  Schedules 


Figure  2-1  shows  the  executed  schedule  for  the  Option  year. 


ID 

TaaK  Name 

Stan 

Finish 

1 

Base  Year 

Fn  8/14/01 

Fn  8/14/01 

2 

Option  1 

Mon  8/17/09 

Fri  8/20/10 

3 

Demonstration*  &  TIMS 

Mon  8/17/09 

Mon  8/9/10 

4 

Customer  Demos 

Mon  8/17/09 

Mon  8/9/10 

5 

DAO 

Mon  8/17/0? 

Wed  9/30/0' 

6 

lSR-to-C2 

Mon  8/1 7/01 

Wed  9/30/0' 

7 

EC10 

Wed  7/14/11 

Mon  8/9/1  < 

8 

Technical  Interchange  Meeting  A  IPTs 

Fri  1/15/10 

Wed  8/4/1 C 

24 

Systems  Engineering 

Tue  9/1/09 

Mon  8/21/10 

25 

Archlecture 

Tue  9/1/09|  Wed  3/24/1  ( 

29 

Integration  &  Test 

Mon  1/4/V0 

Mon  6/21/10 

35 

Develop  me  rt 

Mon  8/17/09 

Fri  8/20/10 

36 

Ontologies 

Mon  8/1 7/09 

Sat  7/1 7/K 

37 

Data  Usefulness  Assessment 

|  Mon  11/2/09! 

Wad  1/13/1  ( 

44 

Geo  Qjftjral  Phase  1 

Mon  8/17/09 

Fri  2/19/10 

.1/ 

Geo  Curtural  Phase  2 

Tue  4/8/10 

Mon  6/28/10 

53 

Anthropological  Instantiate 

Fri  1/1 5/1 0| 

Thu  3/11/10 

58 

Anthropological  Update 

Tue  4/8/10 

Sat  7/1 7/1< 

62 

Fusion 

Mon  1/4/10' 

Wad  8/11/K 

63 

Expanded  Infs  fencing  Phase  1 

Mon  1/4/10 

Wad  3/17/1  < 

69 

Expanded  Infsrsndng  Phase  2 

Tue  4/6/10! 

Wad  8/11/K 

75 

liter  faces 

Mon  1/4/10, 

Fri  8/20/10 

100 

Program  Management 

Mon  8/17/09 

Fri  8/13/10 

101 

Program  Manage  ms  it 

Mon  8/17/09! 

Fri  8/13/10 

112 

Finance 

Mon  8/17/01 

Frt  8/13/11 

113 

Contracts 

Mon  8/17/0? 

Fri  8/13/11 

114 

Sub  Contracts 

Mon  8/1 7/01 

Frt  8/13/11 

115 

Final  Report 

Mon  8/9/1 C 

Fri  8/13/It 

2009 

Qt  3.2009 

Ot  4.2009 

at  1.2010 

Ot 2.2010 

Qt  3.2010 

May  I  Jun 

Jul  |  Aug  1  Sep 

Oct  j  Nov  |  Ooc 

J«i  jFehT Mor 

Apr  |  May  |  Jun 

Jul  1  Auo 

Figure  2-1:  Option  Year  Schedule 


Table  2-1  summarizes  statused  milestones  for  the  program  Option  year. 


Technical 

Baseline  Compieteion 

Revised/Planned 

Actual  Completion 

Milestones/T  asks 

Date 

Completion  Date 

Date 

Customer  Demos 

JUL  10 

AUG  10 

AUG  10 

TIMS 

MAR  10,  AUG  10 

MAR  10,  AUG  10 

— 

Architecture 

MARIO 

MARIO 

MAY  10 

Integration  and  Test 

JUN  10 

JUL  10 

JUL  10 

Ontologies 

JUL  10 

JULIO 

JUN  10 

Fusion/Inferencing 

AUG  10 

AUG  10 

JUL  10 

Interfaces 

AUG  10 

AUG  10 

JUL  10 

Table  2-1:  Statused  Option  Year  Milestones 

TIMs:  No  Technical  Interchange  Meetings  occurred  in  the  Option  Year,  however,  Textron  Systems  par¬ 
ticipated  in  four  (4)  IPTs  in  preparation  for  the  EC  10  demonstration:  Ontology,  Architecture,  Analysis, 
and  ISR-to-C2  and  in  the  EC  10  Demonstration  itself.. 

Architecture :  Completion  of  this  task  was  delayed  as  the  ONR  team  (including  performers)  determined 
the  EC  10  architecture. 

Ontologies:  Since  moving  to  an  Android  platfonn  resulted  in  us  moving  toward  a  database  driven  archi¬ 
tecture  rather  than  ontology  driven  architecture,  thus  work  in  this  area  finished  early. 
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Fusion/Inf erencing:  The  basic  rule  contructs  completed  in  July,  most  detailed  rules  were  written  as  we 
learned  about  the  EC  10  scenario. 

Interfaces:  System  integration  culminated  in  an  integration  exercise  at  Raytheon  in  July. 


2.2  Meetings  Summary 

The  Option  year  program  had  a  defined  set  of  periodic  meetings  that  are  all  documented  with 
meeting  minutes.  These  meetings  included: 

•  Weekly  Team  meetings  with  subcontractors 

•  Monthly  program  review  with  Textron  Senior  management 

•  Monthly  Teleconference  with  ONR,  through  December  2009.  Minutes  have  been  distrib¬ 
uted  to  ONR. 

2.3  Documents 

Reporting 

•  Monthly  Spend  Report  9/2009  -  8/2010 

•  Monthly  Financial  Report  9/2009  -  8/2010 

•  Monthly  Internal  Review  Slides  and  Meeting  Minutes  9/2009  -  7/2010 

Contractual 

•  Textron  /ONR  SOW  TS-W8C03  (AV04)  REV  A  -  1 4  December  2009 

•  Contract  MOD  4-P00003-03  November  2009 

Deliverables 

•  Textron  Option  Year  Final  Report,  TUS-FIN-0002,  20  August  2010 
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3  Option  Year  Technical  Overview 

The  concept  behind  the  Textron  Geo-Cultural  Service  has  remained  the  same  from  Base  year  to 
Option  year,  however,  the  implementation  platform  changed  and  thus  we  were  able  to  add  addi¬ 
tional  features  and  interfaces  with  other  performers  to  extend  the  capabilities  provided  to  the 
ONR  enterprise  and  Marine  user. 

The  option  year  began  with  the  ISR-C2  exercise  in  Hawaii,  the  assessment  of  which  led  to  two 
areas  of  work.  First,  we  needed  to  better  focus  the  types  of  data  we  were  assessing  for  the  War¬ 
fighter  as  there  is  a  lot  of  ‘who  cares’  data.  A  series  of  questionairres  were  developed  in  support 
of  this.  The  Android  was  also  brought  forward  as  a  desireable  platform  for  Textron's  software, 
by  bringing  capabilities  directly  to  the  Marine. 

The  Option  year  architecture  focuses  on  using  the  Android  Smartphone  Operating  System,  via 
the  Motorola  Droid,  and  being  able  to  alert  the  Marine  of  Geo-Cultural  anomolies.  The  android 
platform  as  demonstrated  at  EC  10  provides  a  variety  of  user  interfaces  and  ways  for  the  Marine 
to  access  geocultural  and  anthropological  data  and  inference  via  applications  running  on  it  in 
real-time.  The  solution  also  provides  the  capability  to  capture  observations  and  multimedia  to 
distribute  to  peer  Android  users  and  other  consumers/performers  in  classified  and  unclassified 
space.  This  functionality  was  developed,  integrated  and  tested,  but  not  all  of  it  demonstrated  at 
EC  10. 

The  new  focus  on  Android  brought  with  it  several  questions  of  how  the  Warfighter  would  receive 
the  device  and  how  the  capability  was  presented.  To  help  us  better  understand  what  might  and 
might  not  work  in  terms  of  useful  interfaces,  we  leveraged  Textron  military  and  retired  military 
personnel  whenever  possible  to  get  feedback  on  functionality  and  utility.  EC  10  provided  an  even 
better  opportunity  to  get  user  feedback  from  active  duty  personnel  for  future  work  in  this  area. 

3.1  ISR-C2  Demo  Overview 

The  ISR-C2  demo  at  the  MEC  in  Kaneohe,  Hawaii  in  September  2009  provided  the  first  opportunity  for 
the  Textron  team  to  demonstrate  its  Ontology  basded  anthropological  and  geocultural  rules  based  infer- 
encing  technology.  The  initial  architecture  was  developed  to  provide  a  queriable  backend  system  to  sup¬ 
port  the  Marine  in  real-time,  providing  the  ability  to  assess  threat  significance  of  certain  observation  and 
activities.  Textron  integrated  with  Raytheons  DKKN  and  leveraged  data  from  various  sources  to  provide 
initial  demonstrable  capabilities.  The  architecture  for  the  exercise  has  already  been  presented  in  Textron's 
Base  Year  final  report.  A  lot  was  learned  in  the  exercise  in  terms  of  integrating  with  other  performers  and 
becoming  part  of  the  ONR  TUS/LTSN  enterprise  network.  There  were  two  critical  lessons  learned  that 
dictated  what  was  done  for  the  remainder  of  the  Option  Year  1  effort. 

In  general  there  is  a  lot  of  cultural  and  anthropological  data  available  to  inundate  the  modem  Marine. 
There  are  several  problems  with  it.  First,  a  lot  of  it  is  not  terribly  useful  in  a  tactical  environment  or  for  a 
Marine  on  patrol.  Perhaps  a  planner,  an  analyst  or  Commander  would  have  a  much  different  perspective. 
Second,  cultural  and  anthropolocgical  data  is  very  hard  to  present  effectively  or  provide  in  a  useful  fash¬ 
ion  to  a  consumer  at  any  echelon.  Thus  the  first  thrust  for  Textron  was  to  focus  on  understanding  these 
issues  better  and  apply  them  to  the  Marine  on  the  street,  our  first  subject. 
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Much  of  the  ISR-C2  field  effort  was  plagued  by  communications  issues.  This  is  reality  in  a  tactical  and 
especially  urban  environment;  not  something  to  complain  about  but  something  to  adapt  to.  In  the  exercise 
outbrief,  it  was  suggested  that  we  put  our  applications  and  tools  directly  in  the  Marines  hand  by  adapting 
our  work  to  smartphone  platforms.  We  instilled  requirements  on  ourselves  that  we  be  able  to  exercise 
80%  of  our  functionality  without  any  communications  or  network  and  that  we  do  it  on  an  Android  plat¬ 
form. 

3.2  Data  Assessment 

Comments  from  the  base  year  contract  and  the  ISR-C2  demo,  led  us  to  begin  the  next  phase  by 
assessing  what  data  would  be  most  critical  or  useful  for  the  Marine  user.  Starting  with  the  GIRH 
(5/5/2009),  Textron  added  cultural  and  anthropological  elements  to  develop  a  questionairre  for 
assessing  usefulness  of  various  types  of  data.  Eight  (8)  different  users  representing  the  Marine 
Corps,  Special  Forces,  the  Army  and  the  Navy  Seals  participated  in  completing  two  levels  of 
questionaires  (results  of  the  first  allowed  further  drill  down  with  a  second).  Table  3-1  provides  a 
high  level  summary  of  those  elements  considered  most  useful  to  a  Warfighter.  The  higher  the 
utility  number,  the  more  important  it  is. 


Table  3-1:  End  User  Questionnaire  Results 


ID 

Data  Category 

Data 

Utility  to 
Warfighter 

URBANAREA 

1 

Issues 

Political  Climate 

5.0 

2 

Key  Figures 

Government 

9.0 

3 

Military 

7.0 

4 

Opposition 

9.0 

5 

Tribal 

9.0 

6 

Criminal 

5.0 

7 

American  Sentiment 

Population  demographics 

5.0 

8 

Extremeists 

5.0 

9 

Media 

6.3 

10 

Urban  Geographic  Locations 

Criminal/Gang  neigborhoods 

6.3 

11 

Anti-american  areas 

9.0 

12 

Tribal  neighborhoods 

7.0 

13 

Insurgent  neighborhods 

9.0 

14 

Locations  of  recent  conflict 

9.0 

15 

Social  Traits/Values 

Firearms 

6.3 

16 

Security 

6.3 

17 

Taboos 

Verbal 

7.0 

18 

Gestures 

7.0 

19 

Activities  (e.g.  drinking, proselytizing,  solicita¬ 
tion) 

5.0 

20 

Significant  Dates 

Importance 

6.3 

21 

Non-verbal  communications 

7.0 

22 

Slang 

5.0 

23 

Customs 

Greetings 

5.0 

24 

Negotiating 

5.0 

25 

Manners 

5.0 

26 

Tribal  Clan  Structure 

Basis  of  affiliation  (religious,  political,  crimi- 

5.0 
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ID 

Data  Category 

Data 

Utility  to 
Warfighter 

nal....) 

27 

Boundaries 

7.0 

28 

Biases 

5.0 

29 

Roles  in  conflict 

5.0 

30 

Key  Personnel 

7.0 

31 

Insurgent  Affiliation 

7.0 

32 

Political  Structure 

Insurgent  Affiliation 

9.0 

33 

Key  personnel 

5.0 

34 

Law  Enforcement  Structure 

Protection  of  property,  people 

6.3 

35 

Insurgent  Affiliation 

7.0 

36 

Military  Affiliation 

7.0 

37 

Coalition  Affiliation 

7.0 

38 

Identification 

National:  Passport,  Visa 

7.0 

39 

Military 

7.0 

40 

Driving 

7.0 

41 

Forgery 

7.0 

42 

Habitat/Geography 

Town  layout:  living,  shopping,  economy 

7.0 

43 

Population  Movement  and  migrations 

5.0 

44 

Land  use 

5.0 

45 

Topographic  landmarks 

9.0 

46 

Air/sea/land  customs  proce¬ 
dures 

Bribery 

7.0 

A,!  * 

URBAN:  INFRASTRUCTURES.^. 

f'-sf  If- •-  -*&;  * 

47 

Police  or  military  units  with 
police  authority/mission. 

Locations  and  borders 

7.0 

48 

Structure 

5.0 

49 

Leaders 

7.0 

50 

Communications 

5.0 

51 

Proficiency 

5.0 

52 

Corruption 

5.0 

53 

Patrols 

7.0 

54 

Curfews 

7.0 

55 

Hours 

7.0 

56 

Roads 

Critcal  roads  and  intersections 

5.0 

57 

Usage  patterns 

5.0 

58 

Bridges 

Location 

7.0 

59 

Usage 

5.0 

60 

Alternate  routes 

7.0 

61 

Subterainian  features 

Tunnels 

7.0 

62 

Sewers 

5.0 

63 

Power  Plants 

Significance:  Capacity,  coverage  area 

5.0 

64 

Security 

5.0 

65 

Communications  Facilities 

type  (radio  TV,  satellite,  cell) 

7.0 

66 

location 

7.0 

67 

Security 

7.0 

68 

Vulnerabiliy 

9.0 

69 

Overall  Communications  Ca¬ 
pabilities 

Police 

5.0 

70 

Fire 

5.0 

71 

Military 

5.0 

72 

Insurgent 

9.0 

Use  or  disclosure  of  the  data  contained  on  this  sheet  is  subject  to  the  restriction  on  the  title  page. 

6 


TEXTRON  Defense  Systems^ 

TEXTRON  Systems 


Document  #  TUS-FIN-0002 

Document  Revision 
Document  Date  08/20/10 


10 

Data  Category 

Data 

Utility  to 
Warfighter 

73 

Use  of  infrastructure  to 
mask/camoflage  threats 

Location 

9.0 

SYMBOLS 

74 

Images,  descriptions  and/or 
metadata 

Key  people 

9.0 

75 

Grafitii 

7.0 

76 

Vehicles;  common  types,  cohort  association, 
status  symbols 

7.0 

77 

Cultural  objects 

5.0 

78 

Flags  and  banners 

7.0 

79 

Weapons 

9.0 

80 

Murals/pictures/billboards 

5.0 

81 

signage  or  markers 

5.0 

82 

Age  features,  clothing,  colors,  body  art  and 
jewelry,  uniforms 

5.0 

83 

Clan  features,  clothing,  colors,  body  art  and 
jewelry,  uniforms 

7.0 

84 

Military,  Police  features,  clothing,  colors,  body 
art  and  jewelry,  uniforms 

7.0 

85 

Insurgent  features,  clothing,  colors,  body  art 
and  jewelry,  uniforms 

7.0 

86 

Gang  features,  clothing,  colors,  body  art  and 
jewelry,  uniforms 

7.0 

87 

Group  vehicle  associations 

7.0 

88 

Behaviors/gestures 

Appropriate 

5.0 

89 

Inappropriate 

5.0 

90 

Group  specific  mores  or  behaviors 

5.0 

91 

Displays  of  agression 

7.0 

92 

Sounds 

Background  noise  types  or  descriptions 

7.0 

93 

Human  background  noise:  quiet,  noisy, 
screams.... 

7.0 

94 

Songs/chants 

5.0 

95 

Weapon/explosive  dis- 
charge:normal/abnormal,  types,  times 

9.0 

96 

Dialects,  swears,  key  indicator  words 

7.0 

97 

Temporal:  Cyclical 

Schedules 

5.0 

98 

Seasons 

5.0 

99 

Associated  practices  and  behaviorsfor  each 
of  the  above 

7.0 

100 

Temporal:  Crisis 

Typical  crisis  behaviors  (before  during  after) 

7.0 

101 

Any  group  behaviors/interactions  that  may  be 
triggered  by  a  crisis 

6.0 

102 

Buildings  and  Spaces 

Government 

5.0 

103 

Military 

5.0 

104 

Police 

7.0 

105 

Markets 

5.0 

106 

Commercial 

5.0 

107 

Restaurants 

5.0 

108 

Religious 

5.0 

109 

Education 

5.0 

110 

Factories 

5.0 

111 

City  Infrastructure 

5.0 
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ID 

Data  Category 

Data 

Utility  to 
Warfighter 

112 

Streets 

7.0 

113 

Parks 

5.0 

114 

Symbolism  or  cultural  significance 

7.0 

115 

Clan,  Tribal  affiliations 

9.0 

Additional  assessments  were  done  relative  to  different  phases  of  an  operation  and  various  eche¬ 
lons  consuming  the  data  that  are  not  presented  here,  but  will  likely  have  future  use. 

3.3  Requirements  &  Requirements  Coverage 

The  high-level  technical  requirements  for  the  Option  Year: 

1 .  The  Textron  solution  shall  provide  similar  functionality  as  base  year,  but  should  be  im¬ 
plemented  on  a  Droid :  Geo-cultural  and  anthropology  data  stores  were  made  mission 
specific  to  reduce  the  amount  of  data  on  the  Droid.  There  are  currently  no  inferencing 
engines  available  that  run  on  Android;  all  rules  for  geo-cultural  inferencing  were  devel¬ 
oped  in  standard  JAVA. 

2.  The  Textron  solution  shall  provide  cultural  and  anthropological  lookup  capabilities : 
Various  options  are  provided  in  the  system  for  doing  database  lookups.  Data  can  be 
looked  up  on  a  map  with  events  or  geographically  significant  locations  showing  up  as 
push  pins  on  the  map.  They  can  be  selected  and  assessed  in  detail.  The  VIZ,  augmented 
reality  tool  can  provide  similar  access  to  detailed  data.  Additionally  categorized  data 
with  thumbnail  multimedia  can  be  browsed  and  looked  at  in  detail. 

3 .  The  Textron  solution  shall  provide  data  capture  capabilities  including  multimedia:  Input 
screens  allow  a  user  to  capture  data  in  the  forms  of:  pictures,  audio,  hand  entered  text  and 
guided  questions.  This  data  is  position  and  time  tagged  automatically.  The  majority  of 
the  data  is  needed  for  inferencing,  the  rest  is  needed  to  provide  complete  event  entry  inso¬ 
far  as  it  can  provide  utility  to  the  Warfighter  (i.e.  multimedia). 

4.  The  system  shall  provide  most  of  its  functionality  without  a  network  connection :  All  cul¬ 
tural  inferencing  can  be  done  locally,  with  no  network  connection.  Maps  are  loaded  lo¬ 
cally,  databases  are  updated  and  stored  locally,  and  rules  are  executed  locally. 

5.  Loss  of  network  connectivity  shall  not  impact  performance'.  An  Android  service  was  de¬ 
veloped  to  control  data  flow  to  the  network.  Connectivity  is  tested  prior  to  sending  data. 
Data  is  queued  for  a  configurable  period  of  time  if  network  is  unavailable. 

6.  System  shall  be  integrated  with  the  ONR  enterprise  systems  and  tools  for  2010  demon¬ 
stration:  Connectivity  and  data  flow  for  EC  10  was  demonstrated  by  the  Textron  soft¬ 
ware. 

7.  The  Textron  solution  shall  minimize  network  bandwidth  utilization:  The  current  Textron 
design  does  not  send  multimedia  such  as  images  and  audio  in  the  EntitylD  messages.  In¬ 
stead  URLs  to  the  Android  Webserver  are  used  allowing  a  user  to  reach  back  for  multi- 
media  if  desired.  Feedback  from  EC  10  indicates  a  preference  for  sending  multimedia. 

8.  The  Textron  solution  shall  minimize  user  interaction:  While  keyboard  entry  of  metadata 
is  allowed,  Textron  developed  a  series  of  guided  questions  to  minimize  user  interaction. 
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All  GUI  threads  were  developed  trying  to  minimize  user  activity.  Initial  feedback  from 
EC  10  was  that  more  reduction  is  needed.  Other  feedback  was  that  capabilities  exceeded 
what  was  needed  for  a  reconaisance  Marine;  some  capabilities  were  better  suited  for  the 
CoC  analyst.  It  appears  the  system  needs  to  be  tailored  to  match  different  echelons. 
Voice  control  of  the  Android  is  a  long  term  option  also. 

9.  The  Textron  solution  shall  interface  with  DKKN :  The  system  was  integrated  with  Ray¬ 
theon  DKKN  and  chat  as  well  as  the  ONR  messaging  infrastructure.  Alerts  and  observa¬ 
tions  are  sent  as  EntitylD  messages  embedded  in  ONR  Metadata  messages.  When 
DKKN  receives  them  they  strip  off  the  ONR  Metadata  message.  The  EntitylD  message 
is  then  forwarded  to  the  unclassified  GHUB. 

10.  The  Textron  solutions  shall  get  data  to  GHub  and  across  the  guard  to  the  classified 
GHub:  At  EC  10,  EntitylD  messages  were  placed  in  the  unclassified  GHUB  and  for¬ 
warded  across  the  Guard  and  placed  in  the  classified  GHUB.  There  they  are  available  to 
provide  input  to  other  users  (i.e.  KBSI,  Metron,  and  Chi  Systems  in  the  future)  as  well  as 
trigger  Gumballs  on  the  Tactical  Switchboard  (although  not  confirmed  at  EC10). 

1 1 .  The  Textron  solution  shall  get  data  to  other  Androids :  When  DKKN  receives  an  EntitylD 
message  from  Textron  they  copy  the  Textron  data  fields  into  a  chat  message  and  send  to 
unclassified  chat  users  This  was  developed  and  tested  but  not  demonstrated  at  Empire 
Challenge. 

3.4  Architectures 

Textron  is  buillding  a  piece  of  the  ONR  enterprise  architecture,  not  a  standalone  component.  Thus  our 
architecture  is  dependant  on  what/who  we  interface  to.  The  following  three  section  give  and  overview  of 
the  ONR  arcthiecture,  the  Textron  Architecture  and  the  components  used  to  build  that  architecture. 

3.4.1  The  ONR  Architecture 

The  Textron  applications  or  Android  Apps  reside  in  both  the  Data  Sources  and  the  Application  Services 
layers  of  the  objective  ONR  ISR  architecture  shown  in  Figure  3-1.  The  Android  enables  the  Marine,  as  a 
sensor,  to  record  observations  in  both  scripted  (structured  questions)  and  free-formed  (typed  metadata) 
formats.  Multimedia  such  as  images  and  audio  data  can  also  be  captured  and  stored  in  the  Android  SQL 
database  with  the  observations.  Observations  are  input  to  the  Android  inferencing  App  which  leverages 
local  cultural  and  anthropological  data,  to  be  assessed.  The  App  then  either  generates  an  alert  or  passes 
the  observation  on  to  consumers.  Both  the  Alerts  and  the  Observations  are  sent  through  DKKN  to  the 
Central  Repository  layer.  Alerts  are  sent  to  other  users  in  the  network  via  DKKN  and  are  mapped  along 
with  other  system  alerts  in  DKKN. 
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ISR  Enterprise  Architecture 


Data  Adaptor/ 

Meta  Data 
Generation 

Do  to 

Application  Services 

Process  R Data  to  create 

m"*nrti  tn!  Hat*  Jrt  »h*  f.vm  rtf 
PDF  siatemfrrrfs  which  vstf  be 
stored  r.to  the  Central  Data 

Rnitoluty 

Sensor  Panning  & 

ManprjpmenT 

v 

Data 

Optfrsienw'jy  tvtx>  jmd 
pll>CC**T>Wi. 

Bi 

Automated  J&W 

5  i 

Data  Source 

Data 

Warring  an?  rriicatod 
wtn  a  d*ng*T3us 
co ntfton  o«  iKuaton  is 

rTv/xpriH 

Data  Sources 

Application  Services 

CcntexUd 

ROf 


Contextual 


Central  Knowledge  Store 

Workflow 

Mitps  service-s  ro  quesfiorr. 

Ontology 

C*-$ros  Kny.viedge 

Rules 

Cufcr'l*  tf!  ll«Ol  it  tj 

RDF  Repository 

Sierra  RIP  St3trnv*Tf5 


Alerts  &  Warnings 
Central  Repository 


Cuenoa 


Entity  Wiki 


Person 


Place 


Event 


Tiling 


Group 


Entity  Wiki 


Figure  3-1:  ISR  Enterprise  Architecture 

Embedded  EntitylD  messages  are  the  transport  for  both  message  types  with  DKKN  providing  translation 
services  to  the  GHUB  and  to  chat  messaging,  which  distributes  the  content  of  the  messages  to  the  other 
Androids  and  chat  users,  Figure  3-2. 
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AlertOrObservatlon, 
Location  and  time 
from  EntitylD 
Message 


DKKN  Alert 


GHUB 


SensorlD  field  for  Droids  are  specifically  assigned. 

Entity  ID  field  will  either  say  alert  or  be  a  sequence  number. 
Messages  will  be  posted  in  GHUB,  pushed  across  the  guard 
and  made  available  for  other  performers  on  the  classified 
GHUB. 

Messages  with  certain  EntityNames  will  become 
GUMBALLS  on  tactical  switchboard. 


ONR  Metadata  Message 

Type:  Observation 
Media  Reld:URL  pointing 
back  to  multimedia  web 
server  on 
<comment3> 

Entity  ID 


HTTP  Request:  Web  Server 
Running  on  each  Droid.  Can 
embed  URL  in  AlertOrObserva- 
tion  field  for  multimedia 


Figure  3-2:  Textron  Droid  Data  Flow  within  Enterprise 

DKKN  removes  the  EntitylD  message  payload  from  the  ONR  metadata  message  received  from  the  Tex¬ 
tron  Android  App.  The  message  details  such  as  the  type,  and  specific  content,  along  with  position,  time 
and  possible  multimedia  URL  are  packaged  in  a  chat  message  for  distribution  on  the  unclassified  network 
to  other  Androids  or  hosts.  The  EntitylD  message  is  then  posted  to  GHUB,  pushed  across  the  guard  and 
posted  on  classified  GHUB.  Alert  messages  or  those  with  an  EntityName  “ALERTEX”  (for  general  alert 
or  “ALEKY_FIRSTNAME_LASTNAME”  (for  person  sighting)  will  then  also  intitiate  a  Gumball  on  the 
Tactical  Switchboard. 


3.4.2  Textron  Android  Architecture 


The  Textron  Android  Apps  provide  the  Marine  with: 

■  Network  Independence 

o  Maps  and  databases  local  on  device  (3.5.1  and  3.5.2) 
o  Store  and  Forward  when  network  available 

o  Minimized  use  of  network  bandwidth  (multimedia  provided  upon  request  only) 

*  Scripted  and  freeform  event  entry  with  multimedia  (3.5.3). 

■  Geo-cultural  inferencing  and  look-up  (3.5.2. 1  and  3.5.4). 

■  Map,  augmented  reality  and  list  views  of  normal  events,  Warfighter  observations,  anthropology, 
cultural,  and  geo-data  (3.5.2). 

■  On  board  web  server  provides  multimedia  reach  back  from  desktop/laptop  platforms.  Ultimately 
from  other  handhelds. 

■  Sending  alerts  and  observations  to  applications  and  servers  both  classified  arid  unclassified. 
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To  enable  all  of  this,  several  COTS  and  developed  technolgies  and  tools  were  utilized  in  the  im¬ 
plementation  of  the  Android-based  architecture,  Figure  3-3.  The  compents  of  the  Android  archi¬ 
tecture  are: 

Map  GUI  -  uses  the  Nutiteq  API  to  access,  zoom  and  display  locally  stored  map  levels. 

Location  Service  -  maintains  current  position  information  to  support  data  input  requirements  and 
tracks  user  position  over  time. 

Event  Input  -  provides  not  only  direct  user  input  but  also  time  tagging,  geotagging  and  multime¬ 
dia  input.  Any  inferencing  is  done  here  with  output  generated  to  the  database,  multimedia  store 
and  other  consumers  through  the  message  service. 

Visualization  Service  -  includes  all  the  Android  Activities  such  as  menu  screens,  data  screens  and 
the  augmented  reality  screens. 

Message  Service  -  allows  the  system  to  perform  under  limited  or  no  connectivity  situations  for  all 
data  output.  It  queues  messages  and  tests  periodically  for  connectivity,  sending  data  only  when 
the  network  is  available. 

Database  API  -  used  to  provide  contextual  overlays  on  maps  and  augmented  reality  (Viz). 

SD  Card  -  card  was  used  for  storage  of  all  operational  data.  The  Android  operating  system  im¬ 
poses  numerous  restrictions  on  user  access  and  remote  access  to  the  primary  storage  on  the  de¬ 
vice.  The  SD  card  can  be  easily  accessed  via  USB  from  another  host  and  is  write  accessible  to 
all. 

In  addition  to  those  Activities  and  Services  shown,  a  web  server  runs  on  the  Android  to  provide  URLS  for 
media  retrieval.  The  intent  is  to  minimize  network  bandwidth  utilization  where  possible.  Currently  the 
server  only  works  correctly  from  host  to  Android  not  Android  to  Android.  This  functionality  is  contrained 
based  on  the  COTS  product(s)  being  used,  iJetty. 
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=  Droid  HW;  Blue  =  Textron  Developed  Ul;  Light  Green  =  Textron  Developed  Services; 
Dark  Green  =  Android  Provided  API;  Red  =  3rd  Party  API;  Light  Blue  =  Files 


Figure  3-3:  Android  Architecture 


3.4.3  Architecture  Components 

A  number  of  commercial  off  the  self  components  (hardware  and  software)  were  combined  with 
custom  development  to  realize  the  architecture  for  EC  10.  These  are  briefly  listed  and  discussed 
below: 

COTS  Components 

Android  OS  -  A  Linux  based  OS  with  Google  JAVA  application  development  frame¬ 
work.  Environment  enables  leveraging  of  multiple  apps  within  one  app  and  includes  an 
emulator  for  development.  Version  2.1  was  used  for  this  option  year. 

Motorola  Droid 

■  Arm®  Cortex™  A8  processor  550  mHz 

■  3.7-in.;  WVGA  (480  x  854  pixels);  16:9  widescreen  display 

■  16GB  micro  SD  card  (supports  32GB) 

■  GPS 

■  802.11  b,  g 

■  5Mpixel  camera  with  dual  LED  flash 

■  Multiple  data  input  methods  (keyboard,  touch  screen,  stylus,  camera,  microphone) 
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iJetty  -  Web  server  allowing  reach  back  for  images  and  audio.  This  is  an  on-demand  service  to 
reduce  burden  on  network.  Current  version  does  not  work  phone-to-phone  for  images  but  does 
for  phone-to-desktop/laptop  platforms. 

Wikitude  -  Mobile  augmented  reality  (AR)  browser  that  displays  points  of  interest  (POIs)  on  top 
of  the  phone's  camera  view 
NutiTeq  -  Mobile  mapping  SDK 

SQLlite  -  Cross-platform  public  domain  self-contained,  serverless,  zero-configuration, 
transactional  SQL  database  engine 

Internally  Developed  Software 

System  loading,  configuration,  and  updating:  Software  for  installing  Apps,  providing  custom 
configuration  and  updating  the  database  and  mapping  system 

User  interfaces:  Custom  interfaces  including  map  overlays,  radio  buttons  and  controls,  data  dis¬ 
play  of  combine  metadata  and  mutimedia 

Database  schema:  Schema  for  cultural  data,  anthropological  data,  geographic  data,  event  data, 
key  figures,  tracks  and  multimedia 

Inferencing  framework  and  ruleset:  There  are  no  inferencing  engines  currently  available  for 
the  platform.  A  custom  framework  with  rules  was  developed  in  JAVA 

Location  and  tracking  manager:  time  position  and  time  to  objects  and  track  enitities  ovber  time 

Integrated  combination  of  Android  content  providers,  services  and  activities:  A  working  sys¬ 
tem  required  efficient  implementation  and  integration  of  software  and  capabilities  from  multiple 
sources 

Message  construction  and  queueing:  Build  ONR  metadata  messages,  EntitylD  messages  and 
handle  communications  queuing 

3.5  The  App  Components 

The  app  is  database  driven.  This  section  starts  out  with  an  overview  of  the  database  structure.  Subse¬ 
quent  sections  show  the  databases  realization  as  GUI’s  and  visualization  techniques  in  the  App. 

3.5.1  Database 

To  transition  to  an  Android-based  system,  the  Geo-cultural,  anthropological  and  geo-data  databases  from 
the  base  year  had  to  be  transferred  to  a  Droid.  Those  databases,  as  they  existed  were  quite  large.  To  re¬ 
duce  the  size  of  the  database,  we  chose  to  provide  a  user  with  only  mission-specific  information.  This 
would  allow  them  to  not  only  save  valuable  space  on  the  Android,  but  would  make  the  datasets  small 
enough  that  the  user  can  also  manually  search  through  the  databases  if  desired,  which  extends  the  app  to 
more  than  just  an  inferencing  tool.  Ultimately  full  databases  would  be  kept  only  on  a  wirebased  host  and 
an  Android  would  be  loaded  with  minimal  mission  capabilities  for  a  specific  time  and  place. 

To  extend  the  value  of  the  inference  and  provide  additional  look-up  features,  we  also  created  a  key  fig¬ 
ures,  groups/tribes,  events  and  multimedia  tables  to  leverage  the  onboard  camera  capabilty  of  the  Android 
when  capturing  events  for  inferences.  These  pictures  aren’t  used  in  inferencing,  but  again,  provide  the 
user  with  valuable  lookup  tool  if  stored  with  supporting  data. 

Use  or  disclosure  of  the  data  contained  on  this  sheet  is  subject  to  the  restriction  on  the  title  page. 
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The  database  schema  is  shown  in  Figure  3-4.  The  following  tables  are  the  main  tables  of  the  database. 

Normal  Events  (Geocultural):  The  Geocultural  part  of  the  database  includes  information  pertaining  to 
religious  and  state  holidays,  cyclical  cultural  events,  usage  of  buildings  (including  hours  of  operation), 
and  government  and  political  and  religious  practices.  Many  of  the  entries  are  intended  to  be  referenced  to 
the  Geodata  to  provide  usage  data  on  particular  locations.  Entries  may  also  incorporate  multimedia.  A 
text  field  describes  relevant  infonnation.  Table  3-2  shows  an  eample  entry. 


Table  3-2:  Sample  Normal  Event  (Geo-Cultural)  Table 


ID 

Title 

Metadata 

Multimedia  ID 

Loc  ID 

6 

Eid-al-Fitr 

Islamic  Holiday:  lasts  3  days  in  most  places;  an  offi¬ 
cial  public  holiday  of  1  or  more  days  in  62  countries; 
celebrates  end  of  the  Ramadan  fast;  customarily,  all 
gather  in  festive  clothes.  School,  businesses  closed. 
No  work.  May  witness  prayer  events. 

210 

827 

*Supporting  Tables:  Location,  Multimedia 
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Geodata:  Geodata  defines  locations  and  objects  in  space  and  defines  their  types  such  as  building  types, 
landspace  types  and  structures.  For  EC  10,  Geodata  was  collected  for  buildings  at  the  Battletown  MOUT, 
site  STTW  and  Site  Boston  from  program  provided  Google  Earth  file 

EC10_Green_Devil_vl0_0521  lO.kmz.  Specifically  a  LAT/LON  for  each  comer  of  the  buildings  was  de¬ 
termined.  Ultimately  the  format  should  be  in  MGRS  coordinates. 

*Supporting  Tables:  Geotypes,  Location 


Anthropology:  The  Anthropology  database  contains  various  types  of  anthropology  data  including: 


• 

Animal 

•  Graffitti 

• 

Sounds 

• 

Activities 

•  Migration 

• 

Objects 

• 

Symbols  and  flags 

•  Food 

• 

Gestures 

• 

Vehicles 

•  Gathering 

• 

Speech 

• 

Posters  and  Leaflets 

•  Prayer 

• 

Signs 

Much  of  it  has  associated  multimedia  such  as  pictures  of  graffitti  and  signage  or  examples  of  any  of  the 
entries.  Table  3-3  illustrates  a  typical  entry. 


Table  3-3:  Sample  Anthropology  Entry 


ID 

Type 

Metadata 

Multimedia  ID 

12 

Animals 

Men  on  horseback  fighting.  These  are  features  of 'buzkashi': 
the  Afghan  national  sport,  in  which  skilled  horsemen  must 
snatch  a  headless  goat/cattle  carcass,  circle  the  field  and  drop 
it  in  a  scoring  circle  while  members  of  his  team  assist  and 
members  of  opposite  team  try  to  take  the  carcass  from  him 

17 

*  Supporting  Tables:  Source  Type,  Anthropology  Type,  Multimedia 
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Key  Figures:  The  key  figures  database  contains  images  and  metadata  for  key  figures  as  well  as  associa¬ 
tions  with  the  groups  and  tribes  database.  It  may  include  multiple  associated  images  and  thus  multimedia 
IDs.  Typical  Metadata  could  be:  “An  associate  of  al-Qaida  leader  Osama  bin  Laden  who  was  in  charge 
of  foreign  al- Qaida  militants ,  especially  those  operating  in  Pakistan's  tribal  regions  near  Afghanistan.  ” 
*Supporting  Tables:  Tribal_Clan  (aka  Groups/Tribes),  Multimedia 


Events:  Events  are  divided  into  types  or  Event  Classifications  to  enable  scripted  data  entry  questions  for 
reduced  user  interaction  and  to  assist  in  inferencing  formats.  The  current  types  are  as  follows  with  the 
asterisk  indicating  an  evet  involved  in  inferencing: 

•  Attack/Attempted  Attack*  •  Observed  Object 

•  Cache  Found/Cleared  •  Other 

•  Detainment  •  Person  Sighting* 

•  Disturbance*  •  Propaganda/Graffiti 

•  Gathering*  •  Traffic* 

•  Intimidation/Murder  •  Vehicle  Sighting* 

*Supporting  Tables:  Location,  Multimedia,  Event  Classification 


Tribal_Clan  (aka  Groups/Tribes):  Descriptions  and  affiliations  of  groups  with  metadata.  Multiple  loca¬ 
tions  can  be  associated  to  designate  neighborhoods  or  regions  where  they  have  some  presence 
*Supporting  Table:  Location,  Multimedia 


Multimedia:  Multimedia  itself  does  actually  not  reside  in  the  database.  A  database  table  links  multime¬ 
dia  IDs  to  files  resident  on  the  microSD  card. 

*  Supporting  Table:  Location 


Use  or  disclosure  of  the  data  contained  on  this  sheet  is  subject  to  the  restriction  on  the  title  page. 

16 


TEXTRON  Defense  Systems/*' 

TEXTRON  Systems 


Document#  TUS-FIN-0002 

Document  Revision 
Document  Date  08/20/10 


Anthropology Types 

Sou*  ce  Type 

p  _id:  integer 

p  Jd:  Integer 

Si  Name:  varchar(30) 

S:  Description:  vanchar(230i 

I  Last_Updaced:  dacetime 

9:  Dace Cre*ted  integer 

93  Source_Name:  integer 

91  Source_Description  varchar(230} 

91  LastJJpdated:  dacetime 

Ml  Date Creaced:  datetime 

AndrotdID 


p  Jd:  integer 
Ml  MAC:  text 
91  SENSOR ID:  text 


Key Figu*os 


tfl  Multi  JO:  integer 
91  Fir*t_Name:  varchar(50) 
Last_Name:  varchar(250> 
g;  Metadata:  varchar(250) 
TribeJD:  integer 
fl  Confidence:  floac(50) 

SI  LaatJJpdated:  datetime 
Date_Created:  datetime 
SI  HypeHink:  text 


»  TreckJ>oint 


P  Jd:  integer 
if]  Lit:  float(SO) 

91  Lon:  float(30) 
m  Alt:  float(SO) 

91  Accuracy:  f)oat(30) 
Si  Time:  datetime 
S3  Source:  varehar(50) 
91  TRACKED:  integer 


I 


^Anthropology 


p  _id:  integer 


Tribatjlans 

p td:  integer 

91  Name:  text 

91  Multi_IO:  integer 
^  First^Name^Leader:  varchar<30) 

91  Lasc__Name_Leader:  varchar(lOO) 

Ml  Loc_1D:  integer 

Ml  Political_Aff»liation:  varchar(50) 

97  Metadata'  text 

91  Confidence:  float(50) 

91  Government:  varchar(50) 

91  Military:  varchar(30) 

Ml  LastJJpdated;  dacedme 

M3  Date_Created:  dacetime 

97  hyperlink:  text 

SI  Multi  JD:  integer 
SI  types_id:  Integer 
SI  Metadata:  varchai<230) 
Source_ID:  integer 
Confidence:  flaat(SO) 

SI  LastJJpdated:  datetime 
S3  Oate_Created:  daeetime 
S3  hyperlink:  text 


il  Hul  timed  ie 


P  jd:  integer 


93  MultiJD:  integer 
91  Audio:  varchar 
91  Video:  varchar 
gl  JPEG:  varchar 
S3  Metadata:  varchar(2S0) 
HI  LocJD:  integer 
91  Laxt_Updated:  datetime 
gl  Date Cr«ated:  datetime 


Norma  l E  vents 

NormalJ  ven  t Schedule 

\j  Jd:  integer 

p  Jd:  integer 

9J  Title;  varchar(30) 

91  Event_ID:  integer 

91  Metadata:  varcha*(250) 

91  LocJD:  integer 

£  Multi_IDi  integer 

Ml  Start:  dacedme 

93  Lasc.Updated:  dacetime 

Ml  End:  datetime 

91  Date_Cr«ated:  datetime 

91  Day:  integer 

§§  hyperlink:  text 

Event  J3taJoq CHo«ce» 


p  Jd:  Integer _ 

93  Page_tD:  integer 
9§  Choice:  varchar(30) 

§0  Cholce_Value:  integer 
Ml  Next_Page:  integer 
97  Po*:  integer 


ii  Tiatki 


p  Jd:  integer 
gl  Start_Time:  datetime 
91  End_Time:  datetime 


Even  ^Classification 


p  Jdi  Integer 


91  Classification  Jviame:  varchar(50) 
91  Last_Updated:  datetime 
®  Date Creaced:  datetime 


Events 


p  Jd:  integer 


ID  Date_Time:  datetime 
SI  Metadata:  varchar{230) 

£  LocJD:  integer 
^  Event_Classification  JD:  integer 
£  Multl_ID:  integer 
M3  Last_Updated:  datetime 
ID  Dace^Created:  datetime 
m  tat:  float(50) 

91  Lon:  Roat(SO) 

91  Alt:  Roat(SO) 
hyperlink:  text 
(D  Title:  varchar(50) 

y  t 


jd:  integer 


Type:  integer 
ULj_at:  float(30} 

UU_Lon:  floac(30) 

URJ-at:  flcatfSO) 
UR_Lon:  float(SO) 

LR_Lat:  float(30) 

LR_Lon:  float(SO) 

LL_Lat:  float(50) 

LL_,Lon:  floei(50) 
Date_Time:  datetime 
Confidence:  float(SO) 
Last_Updat*d  datetime 
Date_Creaced:  dacetime 

- X - 


Geode  t* 


p  Jd:  Integer _ 

g]  Geo_Type_lD:  integer 
93  LocJD:  integer 
91  Name:  varchar{50) 

£  Confidence:  float(30) 

91  Last_Updated;  datetime 
K3  Dace_Creaced:  datetime 
S]  MET  AO  AT  A:  text 

1= 


Coo  type* 


p id;  Integer 


SJ  Name:  varchar(50) 

SI  Metadata:  varchar(250) 
SI  Last_Updated:  dacetime 
S)  O  a  ce— Created:  datetime 


Event Oialog Pegee 


p  _id;  integer 


91  Type_IO:  integer 

Question:  varchar(SO) 
§0  Next^Page:  Integer 
Ml  First:  booleen 
93  Type:  integer 
91  Key:  varchar(50} 


Figure  3-4:  Database  Schema 
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3.5.2  Visualization/G  Ill’s 

Three  different  vizualization  techniques  are  used  for  data  or  events. 

1 .  List  views  (used  for  look-ups)  with  image  thumbnails  can  be  used  to  search 
through  topics. 

2.  Map  views  can  be  used  to  look  at  geo-located  data. 

3.  Augmented  reality  is  available  where  the  Android  can  be  used  as  an  orienta¬ 
tion  based  viewer  of  the  immediate  surroundings  with  data  or  events  appear¬ 
ing  as  drillable  icons  in  the  display. 

3.5.2.1  List  Views/Look-ups 

List  views/look-ups  can  be  done  in  several  areas. 

From  the  Cultural  Look-Up  option: 

Symbols  (Geocultural  and  Anthropological):  information  is  listed  by  topic  and  (like 
other  data)  may  or  may  not  include  multimedia. 

Key  Figures  can  include  cultural  figures  from  local  pop  culture,  political  figures,  and 
known/suspected  insurgents,  and  other  persons  of  interest.  The  database  can  be  up¬ 
dated  based  on  observations/events  at  the  end  of  a  mission,  as  it  was  at  EC  10. 

Groups  and  Tribes  similarly,  this  table  contains  local  groups,  tribes,  clans.  Metadata 
about  them  as  well  as  where  there  geographic  boundaries  are,  etc. 

Your  Events  contains  a  list  of  any  data  that  the  user  entered  through  the  Report  Event 
feature. 

From  the  Main  Menu: 

View  Normal  gives  lists  of  either  proximity  and/or  time  based  normal  cultural  activi¬ 
ties  to  understand  immediate  surroundings. 

Figure  3-5  is  a  snapshot  of  the  top  level  menu  all  the  way  down  to  the  fourth  and  fifth  level  of 
detail.  From  the  main  menu,  each  of  the  five  (5)  options  can  be  access  directly  through  the  touch 
screen.  Some  of  the  areas  are  also  connected  internally  with  Maps  showing  events  and  cultural 
information  through  drill  down  and  other  categories  leveraging  the  Maps  for  display. 
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Figure  3-5:  Top=Menus;  Middle=High  Level  List;  Bottom=Detail 


Figure  3-5  illustrates  the  capabilities  under  the  Cultural  Look-up  activity.  The  Key  Figures  activ¬ 
ity  allows  one  to  scroll  through  the  database  of  figures  viewing  thumbnails  and  the  beginning  of 
the  figure  metadata.  Selecting  one  of  the  entries  brings  up  a  full  screen  picture  and  text  display. 
Multiple  pictures  can  be  accessed  and  additional  audio  multimedia  can  be  selected.  The  Groups 
and  Tribes  activity  provides  a  similar  drill  down  of  summaries  and  detailed  descriptions  with 
multimedia.  The  Symbols  activitiy  provides  access  to  the  Anthropology  database  as  described  in 
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3.5.1.  At  the  top  level  it  allows  selection  of  Symbol  type  leading  to  thumbnail  lists  with  summa¬ 
ries  and  to  detailed  information  screens.  Finally  is  the  Your  Events  activity  which  provides  the 
same  type  of  summary  and  detailed  information  with  multimedia  for  accessing  events  captured 
on  the  local  Android.  Follow  on  work  will  be  targeted  at  synchronizing  these  events  between 
multiple  Androids  and  other  applications  or  performers.  Potentially,  the  databases  could  be  sy- 
ched  based  on  unit,  location,  echelon  or  other  interest  group. 

Figure  3-6  shows  the  details  of  the  View  Normal  feature.  As  with  other  list,  there  is  a  top  level 
list  with  the  ability  to  drill  down  for  multimedia,  notes  and  schedule  for  the  activity.  A  time  or 
location  filter  can  be  applied  to  focus  the  list  on  current  location  or  current  day. 
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List  view,  filters  and  detailed  views 
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Figure  3-6:  “What's  Normal?”  Screen  Shots 


3.5.2.2  Maps 

The  mapping  capability  provides  mission  specific  map  information  zoomable  from  level  3  to 
level  17.  All  maps  are  loaded  on  the  Android  prior  to  a  mission  to  eliminate  a  dependancy  on 
internet  connectivity.  Four  diffemet  views  are  available  and  can  be  selected  through  the  menu 
for  the  current  Activity.  Repeatedly  selecting  the  first  opyion  toggles  through  the  map  types, 
Map,  Satellite,  Hybrid  and  Terrain.  Figure  3-7  and  Figure  3-8  show  the  views  and  also  illustrate 
the  map  icons  used  to  indicate  current  location  and  events. 
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Figure  3-7:  Map  Views  and  Types 


Figure  3-8:  Map  Views:  Standard  Map,  Satellite,  Hybrid,  Terrain 


Figure  3-9  illustrates  the  different  overlays  that  can  be  selected.  Currently  the  Events  overlay 
displays  events  on  the  local  Android,  although  objectively  this  will  inlclude  events  from  other 
Androids  and  other  data  sources  such  as  sensors.  The  overlays  are:  Events,  Peers  (other  An¬ 
droids),  Tracks  (where  have  I  and  other  users  been),  Diameters  (define  the  active  event  display 
range),  Alerts  with  a  time  window,  and  Geotypes  (position  and  type  data  from  the  Geodata  data¬ 
base. 


Use  or  disclosure  of  the  data  contained  on  this  sheet  is  subject  to  the  restriction  on  the  title  page. 

21 


TEXTRON  Defense  Systems^ 

TKXTRON  Systems 


Document  #  TUS-FIN-0002 

Document  Revision 
Document  Date  08/20/10 


i&itanA 

V  Sffl]  <29:53  PM 

emplreChaffengc 

Overlays 

Enabled 

y/ 

Events 

Peers 

y/ 

Tracks 

Diameters 

Alerts 

ueoiypes... 

Emplfechallongi? 


HA  K"  AM  i  3:56PM 


Event 


Where  are 
other  Android 
users? 


Building  & 
Infrastructure 


Where  have 
Android  users 
been? 


Figure  3-9:  Map  Overlays 


3.5.2. 3  VIZ:  Augmented  Reality 

The  VIZ  tool  adds  an  augmented  reality  view,  where  geolocated  events  or  object  markers  are 
overlaid  on  a  live  Android  camera  image.  By  aiming  the  camera  at  an  area,  the  image  appears  on 
the  screen  with  “balloon”  boxes  where  events  have  occurred.  One  can  pan  around  the  environ¬ 
ment  and  visualize  where  things  have  happened.  A  compass  is  displayed  to  provide  orientation. 
Selecting  the  balloon  leads  to  detailed  information  about  the  event. 
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Figure  3-10:  Augmented  Reality  Showing  Events  Overlayed  on  Camera  Image 

3.5.3  Data  Entry 

With  all  the  inferencing  taking  place  on  the  Android,  there  needed  to  be  a  formatted  input 
mechanism  for  the  user  to  drive  the  rules.  To  that  end,  we  created  a  front  end  GUI  to  allow  the 
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user  to  enter  his/her  observations  by  generating  free  form  metadata.  This  metadata  is  not  used  for 
inferencing  but  is  saved  with  the  observation  for  reporting.  Each  observation  is  then  qualified 
and  detailed  through  a  series  of  questions  that  are  responded  to  by  touching  radio  buttons.  This 
structured  metadata  is  used  for  inferencing.  Figure  3-11  shows  the  top  level  entry  screen  for 
capturing  metadata  as  well  as  the  audio  metadata  and  imagery  capture  screens.  These  features 


use  the  onboard  Droid  mcrophone  and  camera. 


tjA  y  DHaqiitnw 


Three  tabs: 

1.  General  Information 

2.  Audio  Capture 

3.  Image  Capture 


Defaults  to  current  date  and  time. 
Use  'Edit'  features  if  unable  to  enter  an 
event  observation  at  the  time  it 
occurs. 


Appears  on  every  tab.  User  can 
'submit'  at  any  point  (when  done 
entering  Info,  Audio,  Imagery) 


Figure  3-11:  Report  Event  Screens  -  Main  Input,  Audio  Capture  and  Image  Capture 

As  the  Marine  observes  events  of  interest,  there  is  a  mechanism  for  them  to  enter  the  event  into 
the  database  and  trigger  the  inferencing  rules  to  see  if  that  observation  is  normal,  abnormal,  or 
unknown. 


The  questions  used  to  qualify  the  events  are  based  on  what  geo-cultural  and  anthrolopoligical  character- 
ists  can  help  determine  normality  of  such  an  event.  Table  3-4  illustrates  the  event  categories  and  ques¬ 
tions  developed  for  EC  10.  Touching  next  walks  the  user  through  a  sequence  of  screens. 


Table  3-4:  Event  Entry:  Types  and  Qualifiers 

Event  Type  Attributes 

Attack/Attempted  Attack 

Successful  Attack? 

Yes 
No 

Type 

CF  Engagement 
Direct  Fire 
I  ED  Attack 
IED  Found/Cleared 
Indirect  Fire 
Unexploded  Ordinance 
Unknown  Explosion 

Injuries? 

Yes 

Use  or  disclosure  of  the  data  contained  on  this  sheet  is  subject  to  the  n 
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Event  Type  Attributes 

No 

#  ppl  attacking 
0-10 
10-20 
20-30 
30-40 
40-50 
50+ 


Traffic 


Type  of  arms 

Small  Arms 
Sniper  Rifle 
Machine  Gun 
Rocket 
Mortar 


Type  of  Traffic  Observation 

General  Traffic  Change 
Traffic  Abnormality 

Who’s  participating  in  the  abnormal  behavior? 
Some 
Most 


All 


None 

Traffic  Type  (select  one  or  more  significant  types) 

Cars 

Trucks 

Pedestrian 

Choose  one  or  more... 

Men 

Women 

Children 

Motorcycles 

Other  (Bicycle,  Animal,  Cart) 

None 

Traffic  Volume 
Low 
Medium 
High 

Traffic  Speed  (Relative) 

Slow 

Medium 

Fast 

Traffic  Direction  Movement  (choose  one  or  more  -  to  support  bi-directionality) 

1  North 
South 
East 
West 

2  One  Way  Towards  person/place 
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One  Way  Away  from  person/place 
Bi  Directional 
Origin  or  Endpoint 

Teahouse 

Residence 

Mosque 

Intersection 

Market/Commercial 

Observer/Individual? 

Unknown 

'i'-yOte. Si*  •/  -  i a r  'lii <nisXi'zi ii— i 

Gathering 


Type 

LaborA/Vorking 

Religious 

Meals/Tea 

Shopping 

Celebration 

Demonstration 

Riot 

Kneeling 

Procession  -  line(s)  of  people 
Loitering 

Who's  involved?  (select  one  or  more) 
Men 
Women 
Children 


Size 

Small 

Medium 

Large 

Participation  Level 
Some 
Most 
All 


None 

Location  Description 
Outdoors 
Residence 
Commercial 
Meals/Tea 
Religious  Structure 

Generic  Behavior 
Friendly 
Avoidance 
Hostile 
Passive 
Disinterested 
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Disturbance 

Type 

Harrassment/lntimidation 
Gunfire 
Shouting 
Chanting 
Music 
Riot 
Fighting 

Where? 

Current  Location 
Immediate  Area 
Near 
Distant 

Direction 

North 
South 
East 
West 

Person  Sighting 

Person  #1,  ...#n 

. . .  List  of  key  figures  in  database 
Other 

Name  (if  known) 

Gender 

Male 
Female 

Age 

Child  (<  12  yrs) 
Adolescent  (12-19  yrs) 
Adult  (20+  years) 

What  accessories/belongings  do  they  have?  (Select  1  or  more) 
Tools 
Weapons 
Cell  Phone 
Radio 
Baggage 
What  are  they  doing? 

Kneeling 
Acting  Alone 
Running 
Digging 
Walking 

Meeting  with  Others 
Surveillance 


69  A  v'EICD  AfEBC)6:iiPM 
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Generic  Behavior 
Friendly 
Avoidance 
Hostile 
Passive 
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Add  another  person? 

Yes 

No 

Object  Sighting 

Object  #1,  ...#n 

...view  of  anthro  look-up  screens 
Other 

Free  Text/Audio/Visual  Description 

Add  another  object? 

Yes 

__________  No 

Vehicle  Sighting 

Color 

Make 

Model 

Other  Descriptors 
What  was  it  doling? 

Other 

Free  Text/AudioA/isual  Description 

Cache  Found/Cleared 

Contents  of  Cache 
What  was  done  with  it? 

Description  of  location 
How  they  found  it 

_  Metadata _ _  _ _ 

Intimidation/Murder 

Time  -  if  different  from  time  of  data  entry 

Location  -  if  different  from  current  location 

What  happened?  Type  of  intimidation  (i.e.  harrassment) 

Who  did  it 

Who  was  victim 

Propaganda/Graffiti 
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Event  Type 

Attributes 

Time  -  if  different  from  time  of  data  entry 


Location  -  if  different  from  current  location 


What  was  found 


What  does  it  say/mean? 

Detainment 


Time  of  detainment  -  if  different  from  time  of  data  entry 
Location  of  detainment  -  if  different  from  current  location 
Name  of  detainee 
Description  of  detainee 
Reason  for  detainment 


3.5.4  Rules:  Inferencing  Engine 


During  the  first  year  of  the  TUS  program,  Geocultural  and  Anthropological  event  assessment 
used  an  inference  system  based  upon  the  JBOSS/Drools  engine.  This  software  has  its  own  rules 
language  and  specific  operating  system  requirements.  All  first  year  development  was  performed 
using  PC  laptop  computers  running  the  Microsoft  Windows  operating  system.  Rules  were  writ¬ 
ten  for  this  implementation  and  demonstrated  at  the  TUS  experiment  during  September  2009  at 
the  MEC  in  Kaneohe,  Hawaii. 

Per  guidance  from  Martin  Kruger,  development  moved  toward  an  implementation  that  could  be 
demonstrated  on  a  handheld  smartphone  such  as  an  Android  device  (e.g.,  cell  phone).  The  An¬ 
droid  platform  with  its  linux-based  Android  O/S  was  selected  from  among  the  available  hand¬ 
held  platforms  with  programmable  computational  capabilities.  Assessment  of  available  software 
Apps  for  the  android  was  done  to  locate  an  inferencing  engine  but  none  were  found  although  the 
community  was  interested.  The  android  does  have  sufficient  Java-based  programming  capabili¬ 
ties  to  support  development  of  a  basic  Java-based  logical-chaining  inference  machine  via  func¬ 
tional  blocks  of ‘if-then-else’  rules. 

Storage  and  acess  constraints  led  to  a  decision  to  utilize  two  separate  information  mechanisms  to 
acquire  input  data.  Data  required  to  support  the  inference  engine  functionality  was  derived  from 
the  SQL  database  but  implemented  as  a  parallel  construct.  Another  constraint  was  based  on  the 
dynamic  or  non-dynamic  nature  of  the  SQLlite  database.  Thus  lack  of  state  data  limited  rule  de¬ 
velopment  to  inferences  on  singleton,  atomic  events,  although  a  method  to  inference  on  multiple 
events  that  were  input  within  a  single  ‘time-stamp’  was  later  implemented. 

The  aforementioned  constraints  and  the  experience  (lessons  learned)  during  the  previous  MEC 
experiment  led  to  developing  inferencing  software  centered  around  the  generation  of  a  flexible 
construct  that  would  permit  rapid  generation,  modification,  and  utilization  of  rules  on  the  An¬ 
droid.  The  basic  inference  engine  was  designed  to  fire  rules  that  accept  event  objects  (e.g.,  traffic 
event)  along  with  a  time  stamp  and  location  information  (i.e.,  where  the  event  was  observed). 
Rule-firing  outputs  consisted  of  a  message  containing  status  (normal  or  abnormal),  rationale 
(reason  the  rule  fired),  and  confidence  (e.g.,  high,  medium,  low)  in  the  asserted  inference. 

Another  key  design  aspect  was  the  use  of  rules  as  objects  themselves,  which  enabled  encapsulat¬ 
ing  them  within  a  standard  Java  vector  container.  By  restricting  all  rules  to  utilize  identical  input 
and  output  structures,  rule  sets  could  be  loaded  if/as  required  into  this  vector.  Essentially  the  vec- 
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tor  provides  what  are  effectively  function  pointers  to  rules.  A  simple  iterator  then  operates  over 
all  rules  for  the  specific  input  object  class.  Rules  that  are  not  specific  to  an  object  class  are  thus 
skipped  preventing  erroneous  or  unexpected  results.  After  loading  all  of  the  desired  rules  into  the 
input  vector,  a  driver  method  loops  over  all  input  events  and  passes  each  one  through  the  set  of 
rules.  Output  inferences  are  then  stored  in  a  results  vector  that  is  available  for  output  display. 

Inferences  can  be  viewed  as  logical  correlations  between  existing  information  and  previously 
seen  patterns,  predictions,  and  or  logic-chains.  In  light  of  the  available  event  information,  rules 
were  developed  based  upon  current-time  and/or  proximity  associations,  linkages,  and  known 
geocultural/anthropological  information.  Lacking  the  ability  to  store  and  access  existing  state 
information  (i.e.,  data  from  previous  inputs  and/or  results)  precluded  backward  chaining  and  use 
of  true  cross-event  temporal  inferences. 

Several  base  classes  were  developed  to  support  the  matrix  of  anthropological  and  geocultural 
knowledge  as  well  as  the  available  input  events  to  the  system.  These  base  classes  consisted  of 

Attack 
Disturbance 
Gathering 
Person  Sighting 
Traffic 

Vehicle  Sighting 

Rules  are  described  using  a  simplified  ‘if-then’  construct.  Nominal  times  are  demonstrated  per 
information  specified  for  EC10  experiment  vignettes.  Also  note  that  many  ‘positive’  rules  have 
an  associated  ‘complement’  rule  that  fires  when  conditions  are  outside  of  the  normative  event 
times,  dates,  locations,  etc.  Following  are  some  example  rules  used  at  the  EC  10  demonstration: 

Dog  Fighting  -  If  (shouting  and/or  fighting  is  heard  in  the  immediate  area  on  a  Thursday 
from  3pm  through  4:00pm),  then  it  implies  a  NORMAL  situation  since  a  dog  fight  is 
scheduled  for  this  date/time. 

Buzkashi  -  If  (shouting  and/or  fighting  is  heard  in  the  immediate  area  on  a  Tuesday  from 
2pm  through  3:00pm),  then  it  implies  a  NORMAL  situation  since  a  Buzkashi  match  is 
scheduled  for  this  date/time. 

Harassment  -  If  (harassment  of  local  people  is  observed  in  the  current  location  at  any 
time)  then  this  implies  an  ABNORMAL  situation  since  it  is  indicative  of  potential  insur¬ 
gent  activity. 

Daily  Prayer  -  If  (ALL  observed  men,  women,  and/or  children  are  participating  in  reli¬ 
gious  activities  at  7:00am,  10:00am,  1:00pm,  4:00pm,  7:00pm  Saturday  through  Thurs¬ 
day)  then  this  implies  a  NORMAL  situation  as  these  are  daily  prayer  times  observed  by 
ALL  local  personnel. 

Daily  Prayer  -  If  (NOT  all  observed  men,  women,  and/or  children  are  participating  in  re¬ 
ligious  activities  at  7:00am,  10:00am,  1:00pm,  4:00pm,  7:00pm  Saturday  through  Thurs¬ 
day)  then  this  implies  an  ABNORMAL  situation  as  these  are  daily  prayer  times  should  be 
observed  by  ALL  local  personnel. 
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Tribe-B  Person  Sighted  in  Village  A  -  If  (a  male  from  Village  B  is  observed  in  Village 
A)  then  this  is  an  ABNORMAL  situation  since  the  villagers  do  not  normally  associate 
with  each  other  due  to  mutual  hostilities.  (NOTE:  this  rule  originally  included  a  time- 
frame  filter  that  required  disabling  due  to  the  performance  constraints  of  the  EC  10  vi¬ 
gnettes.) 

Mosque  -  If  (a  general  traffic  change  consisting  of  mostly  people  or  cars  is  observed  tran¬ 
siting  to  the  Mosque  during  Friday  Prayer  Time)  then  this  is  a  NORMAL  situation  since 
everyone  should  be  attending  Mosque  prayers  during  this  timeframe.  (NOTE:  this  rule 
originally  included  a  timeframe  filter  that  required  disabling  due  to  the  performance  con¬ 
straints  of  the  EC  10  vignettes.) 

Pakol  -  If  (a  male  is  observed  in  the  immediate  area  and  is  identified  as  Pakol)  then  this  is 
an  ABNORMAL  situation  since  this  is  an  associate  of  a  known  insurgent.  (NOTE:  this 
rule  originally  included  a  timeframe  filter  and  a  location-specific  filter  that  required  dis¬ 
abling  due  to  the  performance  constraints  of  the  EC  10  vignettes.) 

SUV-618  -  If  (vehicle  color  is  red  and  type  is  SUV  and  license  contains  “618”  and  the 
vehicle  is  observed  in  the  market)  then  this  indicates  an  ABNORMAL  situation  since  this 
vehicle  is  associated  with  known  insurgent  drivers. 

During  the  EC  10  exercise,  we  quickly  learned  the  rule  sets  designed  from  the  original  supposi¬ 
tions  and  vignette  information  were  largely  obviated  since  temporal  information  was  no  longer 
part  of  the  scenario  execution.  In  addition  to  this  change,  location-specific  rules  required  modifi¬ 
cation  and  or  disuse  due  to  the  fact  that  no  specific  building  functions  were  designated.  Despite 
these  changes,  the  rules-generation  framework  worked  well  as  it  permitted  generation  of  new 
rules  in  a  short  period  of  time  as  information  about  the  actual  vignettes  was  revealed  to  our  team. 
The  Marines  were  capable  of  generating  inferences  based  on  their  inputs  after  minimal  training. 
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3.5.5  Output 

The  inferencing  rules  generate  an  output  that  is  either  the  result  of  an  inference,  or  if  no  result  a 
pass  through  of  the  input  data.  Thus  the  result  is  either  normal,  abnormal  or  no  result.  The  re¬ 
sults  are  presented  to  the  Marine,  but  also  meant  for  outside  consumption.  Two  types  of  data  are 
output,  either  and  Alert  or  an  Observation.  The  output  form  is  an  EntitylD  XML  message.  The 
XML  schema  for  the  EntitylD  message  is  shown  in  Figure  3-12.  The  SensorlD  field  is  used  to 
specify  the  messages  as  coming  from  and  Android.  Position  and  time  are  filled  out  by  the  Mes¬ 
sage  Service.  The  EntityName  field  is  set  to  indicate  the  message  type.  If  it  says  Alert  it  indi¬ 
cates  an  alert  to  downstream  applications.  Otherwise  it  is  filled  out  with  a  unique  sequence 
number  generated  by  the  app.  The  AlertOrObservation  field  is  used  to  state  the  basic  results  of 
the  inference  or  a  statement  of  the  observation.  Included  in  here  is  a  URL  for  Multimedia 
Reachback  into  the  Android.  The  BackUpFacts  field  is  used  to  provide  additional  information 
relevant  to  the  event  such  as  the  answers  to  the  questions  that  were  asked  upon  data  input. 

The  EntitylD  message  is  wrapped  in  an  ONR  Metadata  Message  for  transport  to  DKKN.  DKKN 
removes  the  wrapper  and  sends  the  message  to  the  unclassified  GHUB.  It  is  then  transferred 
across  the  guard  to  the  classified  GHUB  for  use  by  other  performers  and  the  Tactical 
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Switchboard.  DKKN  also  removes  the  Android  specific  content  from  the  message  and  sends  it 
out  as  a  chat  message,  including  the  URL.  Again,  this  functionality  was  tested  during  integration 
at  Raytheon  in  July,  but  not  demonstrated  at  EC  10. 
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<?xml  version=" 1 . 0"  encoding="UTF-8,r?> 

El  <ltsn:  EntityldentHessage  xmlns :  ism="urn: us : gov:  ic :  ism:  v2  w  xmlns :  ltsn 
<Security  ism: ownerProducer="USA"  ism: releasableTo=”U3A"  ism: class 


□ 


□ 


<Cor[iimDevice  ID>CommDevice  ID</  CommDevice  ID> 
<SensorID>SensorID</ Sensor ID> 

<Time>2001-12-3 1T12 : 00 : 00</Time> 

<Location> 

<Latitude>0 . 0</Latitude> 

<Lat itudeUncertainty>0 . 0</LatitudeUncertainty> 
<Longitude>0 . 0</Longitude> 

<LongitudeUncertainty>0. 0</LongitudeUncertainty> 
<Altitude>0 . 0</Altitude> 

< Alt it udeUncer taint y>0 . 0</ AltitudeUncertainty> 
<Alt itudeUnits>Meters</ Alt itudeUnits> 

</Location> 

<Ent ityFeatur es> 

< E n t i t yName > E nt i t yName < / E n t i t yName > 
<AssociationID>0</ Assoc iationID> 


wbfa 


<HatchConf idence>0 . 0</ Hat chConf idence> 
<VideQSensorInfo>|,  ^  ]  _ 

<  Aitd  i  oSe  n30  r  M  ic  Ar  r  ay  Inf  o>]TTT| 
<VideoDetectionInf o> . 

<  AudioSensor  Inf  oDOA>] . 

<AudioDetectInfo> . 


< Audio Inf oHfccSegs> . 
<Vo iceTranscr  ipt>[.  ' 


<AudioVideoAssociation> . 
<FacialRecognitionInfo> . 

<L inkedldent>f.  .  .] 
<GaitRecognitionString>|.  .  .| 
<  B  i  ometx  i  c  s  >jTTT| 

<ttl>|TTT[  _ 

<AudioPrint>|.  .  | 

<Thr o ughThe Ua  1  1>).  .  .| 

<Vehic  le  IP>|TTT| 

<Vehic  leD  ismount  Assoc>|T7J 

— -  a - - - - 


f 


<GeoCulturalDroid> 

<AlertOrObservationID>l</ AlertOrObservationID> 

<Aler tOrObservat ion> Alert OrObservat ion</ Alert Or Observat ion> 
<BackUpFacts>BackUpFact3</BackUpFacts> 

</GeoCulturalDroid> 


</ 


<  Re  1  a  t  e  dL  i  nk>http :  //  tamour  i .  orcrc  /  Re  1  at  e  dL  i  nk> 

<CanonicalURI>token</CanonicalURI> 

ltsn: Entity IdentMessage> 


Figure  3-12:  ONR  EntitylD  Message  Schema 
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3.6  Testing 

The  Android  OS  was  a  new  and  different  development  enviroment,  paradigm  and  contained  new 
hardware.  We  had  this  and  other  COTS  tools  to  learn  and  test  before  being  able  to  successfully 
develop  our  system.  When  feasibility  could  be  demonstrated,  software  development  started  with 
a  focus  on  modularity.  Individual  Activities  and  Services  and  base  capabilities  were  developed 
and  tested  independently  and  then  integrated  and  tested  as  a  system.  This  modularity  will  make 
future  enhancements  and  changes  easier  to  implement. 

Enterprise  sytem  interfaces  were  designed  and  developed  early  through  continuous  conversations 
with  Raytheon  to  ensure  we  were  developing  in  a  way  that  suited  the  enterprise.  In  parallel,  we 
developed  and  tested  interfaces  with  DKKN  as  well  as  messaging  constructs  per  the  ONR  guid¬ 
ance.  Although  the  guidance  changed  over  time,  the  plumbing  to  move  data  existed  and  was 
tested. 

System  level  integration  and  testing  was  performed  on  site  at  Raytheon  on  July  19lh  and  20th. 
There,  we  were  able  to  demonstrate  message  construction  within  the  Textron  App,  delivery  to 
DKKN,  message  deconstruction  and  rebuilding  for  the  enterprise  as  EntitylD  messages  and  also 
message  delivery  to  Android  users  via  notifications  and  chat  as  well  as  to  Ghub  users.  As  the 
message  schema  continued  to  change,  we  simply  adjusted  the  schema. 

Testing  and  tweaking  continued  throughout  EC  10,  August  2-13. 

3.7  Demonstration 

Objectives: 

1.  Demonstrate  multiple  ways  in  which  the  app  can  provide  geo-cultural  context  and 
actionable  intelligence  to  the  Warfighter. 

a.  Data  collection  and  inferencing  (with  alerts  to  the  Warfighter) 

b.  Look-ups 

c.  Visualization 

2.  Demonstrate  the  app  working  as  part  of  the  ISR  Enterprise 

a.  Integrated  alerts  with  DKKN:  to  the  CoC  and  other  Android  users 

b.  Multimedia  reachback 

c.  Messages  succesfully  cross  the  guard  and  are  visible  on  Tactical 
Switchboard 

3.  Demonstrate  output  being  consumed  by  other  analysis  tools 

4.  Obtain  feedback  regarding  usability  and  relevance  to  the  Warfighter 
Results:  Corresponding  to  the  items  above... 

1 .  Demonstrate  multiples  ways  in  which  the  app  can  provide  geo-cultural  context 
and  actionable  intelligence  to  the  Warfighter:  Some  methods  of  providing  geo- 
cultural  and  anthropological  context  were  more  used  than  others. 

a.  Data  collection  and  inferencing  (with  alerts  to  the  Warfighter):  Data  col¬ 
lection  was  used  nearly  every  day  to  capure  location  of  symbols,  signs. 
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propaganda,  key  figures,  and  key  events  (such  as  person  sightings,  detain¬ 
ments,  vehicle  sightings,  etc). 


On  Day  2  of  EC10/GDII,  Marines  were  able  to  identify  Hafiz  (key  figure, 
top  left  of  Figure  3-13)  using  the  database  in  the  app.  This  resulted  in  de¬ 
tainment  of  he  and  two  others  as  well  as  discovery  of  a  weapons  cache. 
This  discovery  thwarted  a  potential  attack.  The  personnel  and  vehicle  were 
processed  using  the  app:  photos  and  location,  date  time  and  metadata  were 
captured. 


Figure  3-13:  August  9th  -  Database  Use  and  Data  Entry 

Inferencing  was  demonstrated  in  test  mode  as  well  as  on  day  5,  Friday, 
when  we  were  able  to  script  our  own  mini-scenario.  The  app  alerted  the 
Marines  to  anomolous  ‘traffic  change’  when  the  Marines  ‘saw’  heavy  traf¬ 
fic  going  from  Village  B  to  Village  A,  they  saw  a  normal  ‘gathering’  of 
people  at  the  ‘Mosque’  for  Friday  prayer  and  ‘person  sighting’  anomolies 
when  key  figures  from  Village  B  were  ‘seen’  in  Village  A. 

b.  Look-ups :  Look-ups  were  used  extensively  to  understand  the  meaning  of 
signs  and  symbols.  The  meanings  were  helpful  in  questioning  and  gaining 
intel  from  locals  and  key  figures. 
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Figure  3-14:  August  10th  -  Propaganda  and  Sign  Discernment 


c.  Visualization'.  Although  the  maps  and  augmented  reality  were  very  well  re¬ 
ceived  during  integration  week,  they  weren’t  utilized  much  during  execu¬ 
tion  of  the  scenario.  It  is  believed  this  was  partly  due  to  the  limitations  of 
the  scenario  and  available  data  as  well  as  some  difficulties  in  seeing  the 
screen  in  sunlight. 

2.  Demonstrate  the  app  working  as  part  of  the  ISR  Enterprise'.  Messages  were  suc¬ 
cessfully  sent  from  the  phones,  to  DKKN,  to  GHUB  and  through  the  guard.  Due 
to  network  realiabiltiy  and  congestion,  messages  didn’t  appear  to  be  getting 
through  early  in  the  week.  We  re-validated  that  we  had  the  correct  schema,  and 
Wednesday  we  were  able  to  see  some  messages  and  Thursday,  many.  We  don’t 
have  final  numbers  on  how  many  Marine  generated  messages  made  it  through  be¬ 
cause  some  phones  were  also  generating  test  messages  during  vignettes.  We  have 
to  go  back  and  sort  through  all  messages  to  find  the  final  number. 

a.  Integrated  alerts  with  DKKN:  to  the  CoC  and  other  Android  users:  Using 
the  most  current  filters  on  DKKN  and  the  latest  Textron  EntityName  con¬ 
struct,  we  weren’t  able  to  see  all  alerts  on  DKKN.  However,  mid-week, 
Maj  Filler  was  able  to  send  an  alert  message  from  the  Textron  app,  receiv¬ 
ing  it  less  than  a  minute  later  through  the  DKKN  app.  This  demonstrated 
how  the  two  apps  work  together,  despite  some  minor  filter  and  naming  dif¬ 
ferences. 

b.  Multimedia  reachback:  Reachback  was  preliminarily  tested  at  raytheon 
from  PC  to  android  which  worked,  but  android-to  android  reachback  con¬ 
tained  a  rendering  issue  that  was  unable  to  be  resolved  as  it  was  a  problem 
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far  within  the  source  code  of  iJetty.  Although  the  functionality  was  tested 
from  Droid  to  PC,  it  was  not  used  at  EC  10. 

c.  Messages  succesfully  cross  the  guard  and  are  visible  on  Tactical 

Switchboard :  The  Textron  alerts  and  observations  made  it  across  the  guard 
but  it  is  unknown  and  unlikely  that  our  messages  made  it  all  the  way  to  tac¬ 
tical  switchboard  because  we  did  not  have  the  right  contents  in  EntityName 
field.  It  was  late  in  the  week  before  we  knew  of  this  issue/change.  We 
original  thought  that 6  Alert’  had  to  be  the  EntityName,  but  turns  out  it 
needed  to  contain  an  actual  entity  name  (i.e.  of  a  key  figure). 

3.  Demonstrate  output  being  consumed  by  other  analysis  tools :  As  discussed  al¬ 
ready,  we  succesfully  integrated  with  Raytheon’s  DKKN,  both  the  ‘regular’  and 
app  versions.  Early  on  in  the  development  cycle,  we  talked  some  with  Metron, 
Chi,  Los  Alamos  and  Aptima  about  consuming  our  output.  We  went  so  far  as 
sending  them  sample  output,  but  in  the  crunch  to  prepare  for  the  demonstration,  I 
don’t  know  if  any  of  these  performers  actually  implemented  comsumption  of  our 
output. 

4.  Obtain  feedback  regarding  usability  and  relevance  top  the  Warfighter.  We  re¬ 
ceived  invaluable  feedback  from  the  Marines  on  Friday,  August  13.  We  spent  ap¬ 
proximately  1  hour  with  them,  simply  learning  about  their  roles  as  recon  Marines 
and  analysts.  From  that  discussion  we  gleaned  a  lot  of  next  steps,  see  section  4. 
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Scenario  Development  We  worked  early  and  often  with  Kelli  Kuduk  to  try  and  ensure  we  had  a 
Nawa-esque  scenario  that  would  utilize  our  Nawa-focused  geo-cultural  and  anthropological  data 
stores.  Unfortunately,  given  the  limited  actors  and  resources,  much  of  what  we  discussed  was 
not  used,  or  was  acted  out  in  a  limited  way.  We  also  had  little  information  on  building 
names/numbers,  locations,  usages. 

Despite  these  challenges,  we  were  able  to  adapt  and  present  more  capability  than  in  the  ISR-C2 
demo  in  Hawaii  last  year. 
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4  Results,  Conclusions  and  Next  Steps 

Overall,  Textron  had  a  good  showing  at  EC  10.  We  were  able  to  show  utility  of  the  various  parts 
of  our  tool  that  provide  geo-cultural  context  to  the  Warfighter.  We  also  got  a  lot  of  invaluable 
feedback  from  the  Marines.  Additionaly  we  were  able  to  make  a  lot  of  our  own  obeservations  on 
functionality  and  utility  when  accompanying  the  Marines  in  the  field.  We  have  nine  (9)  key  take 
aways  that  may  fuel  our  next  steps: 

1.  Inferencing  should  be  separated  from  the  look-up  and  data  entry  tools. 

On  the  final  day,  we  spent  some  quality  time  with  the  Marines,  learning  of  their  various 
Warfighter  roles  and  responsibilities.  We  learned  that  the  Recon  Marine  simply  observes 
and  does  not  act  unless  told  to.  They  can’t  act  on  an  abnormal  inference  and  therefore  in¬ 
ferencing  functionality  is  better  suited  for  an  analyst.  However,  both  the  Major  and  Lieu¬ 
tenant  commanding  the  exercise  indicated  this  functionality  would  be  of  importance  to 
them  if  fully  matured.  The  inferencing  capabilities  were  appreciated  by  the  Analysts. 
Recommendations  from  several  of  the  Marines  indicated  that  the  software  capabilities 
should  be  targeted  at  multiple  echelon  levels  (e.g.,  low-level  and  high-level  inferencing). 
From  several  discussions  with  the  Marines  during  the  course  of  the  experiment  it  is  fairly 
obvious  that  the  functionality  should  be  separated  (maps,  Vis,  inference,  geocultural  data 
lookup)  and  each  tool  in  our  software  system  refined  for  the  specific  mission  tasking  and 
command  level.  Some  discussions  with  ONR  have  hinted  at  a  new  Strategic  Corporal 
type  mindset,  where  an  enhanced  capability  to  the  Marine  could  modify  his  ability  to  re¬ 
act  to  situations  and  understand  his  environment.  A  question  for  ONR  here  then  is  are  we 
trying  to  support  the  current  force  Marine  or  are  we  trying  to  define  the  future  force  ma¬ 
rine  by  enhancing  his  ability  to  act. 

2.  Data  entry  needs  to  be  simplified. 

The  Recon  Marine  would  prefer  desktop-level  input  screens  or  a  default  menu:  text,  au¬ 
dio,  imagery  that  are  tied  together  and/or  who/what/where/when  or  SALUT-based  entry 
fields.  Remove  the  Q&A  prompting  that  is  currently  needed  for  inferencing.  Anything 
that  prevents  them  from  maintaining  control  of  their  weapons  at  all  times  could  easily 
make  them  vulnerable  to  attack.  Some  mechanism  to  automate  the  input  and  output  proc¬ 
ess  (e.g.,  voice  recognition)  would  be  extremely  helpful  here. 

3.  We  should  look  into  inferencing  on  sensor  output. 

This  is  complementary  to  #1.  This  year,  we  developed  an  inferencing  tool  that  inferences 
on  Marine  input.  After  EC  10,  however,  we  understand  that  there  are  sensor  systems  that 
could  give  us  more  relevant  input  for  inferencing  and  would  also  lessen  the  burden  on  the 
Warfighter  (who  stated  that  they  want  something  that  is  three  ‘clicks’  or  less... more  than 
that  is  too  much). 

Inferencing  on  sensor  input  would  however  give  us  dependence  on  the  network.  TAVI 
and  TINDER  are  two  sensor  providers  that  stood  out  as  being  able  to  provide  us  with  in¬ 
put  on  which  we  could  inference.  Traffic  patters,  numbers  of  people  in  a  place  at  a  cer¬ 
tain  time,  etc. 

4.  Look-ups,  integrated  maps  and  augmented  reality  were  well-received. 
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Look-ups  are  a  keeper!  They  were  the  most  utlized  portion  of  the  tool. 

The  Marines  and  DV’s  really  liked  the  mapping  and  augmented  reality  features;  it  would 
be  a  good  idea  to  look  into  how  we  can  make  these  more  usable  features.  They  were  not 
used  in  EC  10,  perhaps  due  to  scenario  limitations. 

5.  Use  of  state  information. 

State  information  is  extremely  important  in  providing  inferencing  capabilities  and  should 
be  a  requisite  part  of  any  further  development  efforts.  This  will  provide  the  ability  to  cor¬ 
relate  multiple  events  over  time  that  are  necessary  to  actually  detect  patterns  and  produce 
associated  forward/backward  chain  inferences. 

6.  Viz  tool  and  other  form  factors. 

The  augmented  reality  tool  displaying  the  relative  locations  of  prior  events  made  an  im¬ 
mediate  positive  impact  and  was  seen  as  a  viable  tool  for  ground-level  and  vehicular  use. 
Regular  (periodic  or  event-based)  updating  of  the  databases  would  be  required  and  a  dif¬ 
ferent  form-factor  adopted  that  will  provide  additional  viewing  capability,  especially  in 
difficult  lighting  conditions.  A  side  benefit  of  this  is  that  the  larger  form-factor  should  re¬ 
duce  or  eliminate  many  of  the  GUI  and  battery  constraints  that  currently  exist  with  the 
hand-held  devices. 

7.  Android  connectivity  and  cohabitation. 

The  initially  poor  Wi-Fi  reception  of  the  Motorola  Milestone  (ONR-specified)  Androids 
severely  impacted  further  use  of  the  Android  tool.  It  also  became  quickly  apparent  that 
sharing  application  space  with  other  participants  led  to  confusion  as  to  which  application 
to  use  to  generate  and  report  which  type  of  event  had  been  observed.  Some  functionality 
(e.g.,  mapping)  overlapped  with  other  vendor’s  application  capabilities.  The  Textron  ap¬ 
plication  did  not  require  constant  Wi-Fi  communications  and  was  still  useful  when  the 
network  was  not.  When  the  other  applications  failed  to  communicate  with  the  command 
center,  there  was  a  major  tendency  for  the  Marines  to  cease  use  of  the  Android  device  en¬ 
tirely  to  the  detriment  of  the  Textron  application.  Later,  during  the  second  week  of  the 
experiment,  a  new  set  of  Samsung  Intercept  Android  phones  were  provided  to  the  Ma¬ 
rines  for  trial.  Some  communications  issues  still  persisted,  however,  and  it  wasn’t  until 
the  final  day  of  the  experimental  vignettes  that  full  functionality  could  be  demonstrated. 
The  previous  events  had  a  prejudicing,  albeit  realistic,  effect  on  the  resulting  impressions 
of  the  software. 

8.  The  Geo-Cultural  support  of  this  tool  is  most  useful  in  the  first  couple  weeks  of  a 
deployment. 

Generally,  after  a  week  or  so,  the  Warfighter  has  a  good  sense  of  what  is/isn’t  normal. 

This  tool  gives  them  a  resource  for  the  time  when  they  are  learning.  They  can  carry  the 
knowledge  with  them  and  not  have  to  try  to  retain  everything  they  learn  in  their  training. 

9.  Data  Synching: 

If  we  are  going  to  continue  providing  the  user  with  look-ups  and  data  entry,  we  need  a 
way  to  synch  the  data  on  all  the  Androids  so  that  everyone  has  a  complete  and  current 
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data  set.  Rudementary  data  synching  was  done  manually  on  the  Key  Figures  database  at 
EC  10.  This  should  be  automated  in  the  future. 

10.  Verbal  Translation. 

Verbal  translation  was  greatly  desired  so  that  Marine  could  understand  what  a  person  was 
saying  if  they  did  not  have  a  human  translator  with  them.  Processing  time  needs  to  be 
quick.  An  initial  version  of  the  Google  Voice  Actions  app  is  available  on  the  new  Droid  2 
platform  released  in  early  August.  This  multilanguage  voice  app  is  objectively  planned 
as  a  language  translator. 

11.  Network  and  Access  Security. 

If  security  is  needed  on  Droid,  Marine  comment  was  to  use  fingerprint  as  pass  code  so 
that  Marine  does  not  have  to  type  or  remember  another  pass  word.  Textron  is  currently 
investigating  Android  security  issues  as  an  internally  funded  effort,  both  in  terms  of  user 
access  and  platform  access  to  distrbuted  resources. 

12.  Tribe/Group/Clan  Neighborhood  Overlays. 

Develop  “roaming  net”  for  groups  (tribal,  ethnic,  racial,  etc.)  for  map  display  that  will  in¬ 
form  user  of  the  area  said  group  is  normally/typically  found  to  move  around  in.  This 
could  help  raise  alerts  if  members  of  one  group  are  found  outside  their  typical  area.  For 
example,  in  the  vignettes,  Village  B  did  not  leave  their  village  nor  travel  much  in  the 
area,  so  it  was  important  to  note  when  one  of  their  members  did  leave  the  area.  A  map 
based  application  using  simple  spatial  analysis  could  inform  user  when  a  person  identi¬ 
fied  from  one  place  is  found  outside  of  their  typical  “roaming  territory”  and  aid  in  intel. 
Working  with  Chi  Systems  to  leverage  their  map  overlays  woiuld  be  a  desireable  but  dif¬ 
ficult  (security)  way  to  proceed. 

13.  Alternate  Data  Display  Approaches. 

We  might  consider  displaying  normal  events  as  a  Calendar  Schedule  (similar  to  what  you 
get  with  your  Microsoft  Outlook)  with  look-ups  by  date/hour. 

14.  Event  Positioning. 

Current  system  reports  position  of  Android  when  making  an  observation.  Need  mecha¬ 
nism,  perhaps  map  based  to  capture  the  position  of  an  event  or  sighting  if  not  the  same  as 
current  user  location. 

15.  Data  Tailoring. 

Need  to  tailor  information  delivered  to  Marine  in  the  field  by  Commander’s  Intent,  loca¬ 
tion,  time  of  year  and  current  political  environment.  Different  anthropological/geo- 
cultural  data  is  needed  under  different  circumastances  or  environments. 

16.  Visual  Search. 

Akin  to  Google  Goggles  wherein  marine  takes  picture  of  person,  vehicle,  object,  activity, 
etc.  and  says  “search”  and  application  tells  him/her  what  or  who  they  are  looking  at  and 
anything  else  they  may  need  to  know  (HVP,  abnormal  activity,  etc). 
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This  short  list  of  lessons  from  EC  10  will  help  balance  trade  studies  in  the  upcoming  option  year 
effort.  Of  most  value  in  this  exercise  was  getting  the  direct  user  feedback  both  functionally  and 
doctrinally  on  all  efforts  to  date. 
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5  Acronym  List 

Table  5-1:  Acronym  List 


API 

Application  Programming  Interface;  Application  Program  Interface;  Application  Programmer 
Interface 

C2 

Command  and  Control 

CLOC 

Company  Level  Operations  Center 

DKKN 

Distributed  Knowledge  and  Knowledge  Needs 

EC10 

Empire  Challenge  2010 

ERDC 

Engineering  Research  and  Development  Center  -  US  Army  Corp  of  Eng. 

GDII 

Green  Devil  II 

GIRH 

Generic  Intelligence  Requirement  Handbook 

GUI 

Graphical  User  Interface 

ISR 

Intelligence  Surveillance  and  Reconnaissance 

RBIE 

Rules  Based  Inferencing  Engine 

RDF 

Resource  Description  Framework 

SME 

Subject  Matter  Experts 

SOAP 

Simple  Object  Access  Protocol 

TIM 

Technical  Interchange  Meeting 

TUS 

Transparent  Urban  Structures 

XML 

Extensible  Markup  Language 
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